Psychometric properties of the Providers Survey in the Brazilian context of mental health: a validation study

ABSTRACT BACKGROUND: Precisely determining the aspects related to an instrument's validity and reliability measures allows for greater assurance of the quality of the results. OBJECTIVES: To analyze the psychometric properties of The Providers Survey in the Brazilian context of mental health services. DESIGN AND SETTING: The instrument validation study was conducted in Montes Claros, Minas Gerais, Brazil. METHODS: The validation study was conducted using the Consensus-based Standards for the Selection of Health Measurement Instruments checklist to analyze its validity and reliability. RESULTS: A committee of expert judges performed content validation after which the Content Validity Index was calculated. Construct validation took place through Exploratory Factor Analysis using the Kaiser-Meyer-Olkin Test criterion and Bartlett's Sphericity Test. Reliability was verified using test-retest reliability. The significance level adopted for the statistical tests was 5% (P < 0.05). The final instrument comprised 54 questions. The Content Validity Index was 97%. Exploratory Factor Analysis identified a Kaiser-Meyer-Olkin index of 0.901 and Bartlett's Sphericity Test with P < 0.001. We obtained a Cronbach's alpha coefficient of 0.95 and an intraclass correlation coefficient of 0.849. CONCLUSIONS: The Providers Survey, translated and adapted into Portuguese, was named the Work Assessment Instrument for the Recovery of Mental Health. It presented adequate psychometric properties for evaluating work-related practices for the recovery of psychosocial care network users.


INTRODUCTION
The mental health policy in Brazil has shifted from a clinical model with an emphasis on reducing or removing mental health symptoms to a broader understanding based on an active, nonlinear ongoing journey that involves rebuilding oneself and living a full and meaningful life.
To this end, the concept of health is intimately disconnected from the absence of linked diseases, and harmonization exists between all conditioning factors and determinants of health, such as food, housing, leisure, safety, and work. The weight of these factors provides an individual with health from a biopsychosocial perspective. 1,2 The inclusion of patients with mental disorders in daily and social activities began in Brazil in the 1970s and in the United States and in other European countries in the mid-1980s, and the use of the Recovery concept expanded. The expansion of this concept took place through the mobilization of users, family members, professionals, and managers in favor of actions that would provide an optimistic model of personal power to users of Mental Health services, thus consolidating a set of elements that empowered them to redirect their lives after being diagnosed with a mental disorder. 3 One proposal for the implementation of Recovery for people with mental disorders is their insertion into the labor market, as work can act as a support component in the reinstatement of these individuals. 4,5 Studies indicate that the inclusion of patients with mental disorders in the labor market has a positive economic, psychosocial, and clinical impact on their lives. 6,7 Another important point is that employment correlates with short-term reductions in mental health costs. 8,9 In 2016, in the State of Connecticut in the United States, The Providers Survey instrument was developed to verify the perception of job providers regarding the relationship between work and the recovery of people with mental disorders. 10 There is a lack of evaluative instruments regarding the proposals of Recovery After validating the instrument, we verified its reliability through Internal Consistency analysis and its stability through test-retest.

Content validation
The content validation stage of the Providers Survey instrument included the participation of a committee of 10 specialist professionals with different activities in the components of the Psychosocial Care Network in the city of Montes Claros, MG, Brazil, who had knowledge of mental health recovery. Of these professionals, two were doctors, one was a nurse, one was a dentist, and six were psychologists, as described in Table 1.
We sent a questionnaire containing The Providers Survey with 74 items to be evaluated by the expert committee. To verify the clarity, scope of the items, and the general structure and layout of the instrument, this questionnaire included a specific field related to each item so that each specialist could put their considerations regarding the item evaluated for further adequacy. Throughout the evaluation process, the responsible team was able to clarify doubts and possible queries.
To conduct the aforementioned assessment of the instrument's items, we used a 4-point Likert scale, in which each item of the instrument received the following classification: 1) being a non-equivalent item; 2) a slightly equivalent item that needs many revisions; 3) an equivalent item that requires few revisions; and 4 a fully equivalent item. Instrument items with a score of 1 or 2 were excluded.
After this step, the instrument presented 69 items, as two questions from the first domain, two questions from the second domain, and one question from the third domain were excluded.

Construct validation
Construct validity, also known as factor validity, is considered one of the most important measures of instrument validity in research and aims to verify whether the items of the instrument have a reliable and adequate representation of the construct to be measured.
At this stage, 318 health professionals with higher education working in cities included in the Regional Health Management of Montes Claros, located in the north of the state of Minas Gerais, Brazil, participated as respondents, as described in Table 2.
The instrument version that also contained the ICF was sent

Reliability
The reliability of the instrument was verified after content and construct validation. We verified the Internal Consistency using the α-Cronbach's coefficient and the stability using the test-retest using the Intraclass Correlation Coefficient.
For the stability analysis, we used the test-retest with a sample of 51 respondents, with an average interval of 12 days after the test.  Table 4.
After analyzing the psychometric properties of The Providers Survey instrument, it is clear that such measures may vary according to changes in the study population and context presented.
It is desirable that new studies be conducted in several Brazilian regions to verify the attitudes and actions of professionals focused on incentive practices and approaches aimed at the recovery of the Psychosocial Care Network service users.

DISCUSSION
We used a committee of expert judges to evaluate the constructs; of this committee, at least five were specialists in the area. 16 Based on the judgment of experts in the field, we verified the degree of equivalence of each item of the instrument under analysis to measure the degree of relevance of each item of the instrument in a given construct.
After the subjective assessment performed by the group of experts, we calculated the Content Validity Index, which measures the percentage of judges who agreed on the aspects of each item of the instrument. 14,15 Instrument items that received a score of 1 or 2 were excluded. 13 An acceptable Content Validity Index must be at least 0.80, and preferably greater than 0.90. 12,14,15,17,18 For construct validation, we used the Factor Analysis strategy, considered one of the most important measures of instrument validity in research that aim to verify if the items of a given instrument are a reliable and adequate representation of the construct to be measured. 14,15 Following the recommendations regarding the constitution of the sample to conduct a Factor Analysis, we aimed for the participation of at least 100 respondents, with 5-10 respondents being ideal for each item of the questionnaire. 11,[19][20][21] To assess the factorial structure of the instrument, we used the Kaiser-Meyer-Olkin index and Bartlett's Sphericity Test with a significance level of P < 0.001, which indicates the suitability of factor analysis in the process. 22 After verifying the suitability of the factor analysis, we rotated the matrix to separate the variables between factors. The varimax rotation method is the most commonly used method in research of this nature, as it treats domains as independent allowing us to exclude questions with a factor loading < 0.5. Studies recommend that the Principal Components should have eigenvalues > 1. [23][24][25] To assess the reliability of an instrument, we used the internal consistency based on Cronbach's alpha coefficient and the stability based on the test-retest using the Intraclass Correlation Coefficient via the Pearson correlation test. 14,15 The test-retest, which allows the reproduction of a result in time and space, provides homogeneity     Hosting the user, their family, and/or companions Recording a complete personal and family anamnesis/history Conducting case follow-up Providing a reference technician for case management Offering counseling and/or guidance or psychotherapy/other psychological interventions Participating in the formulation of the diagnosis Prescribing and/or administering medication Monitoring medication adherence Guiding the use of medications and the effect on the user's psychic condition for the user, their family members, and/or companions Identifying and addressing stigma and discrimination/prejudice Referring users for medical attention and care Referring users to social support services to search for work/employment (legal associations and public services) Providing job search support Referring users for treatment of alcoholism and/or the use of psychoactive substances Providing treatment for alcoholism and/or the use of other psychoactive substances Referring users to the housing support service Providing housing search support Referring users to socialization activities and recreational/leisure support Providing socialization activities and recreational/leisure support Referring users to legal support Involving family members in supporting users in mental distress Providing peer support and guidance and/or a therapeutic companion Providing guidance on the self-care and well-being of users Addressing issues related to forms of violence or forms rights violations. Setting a goal for getting a job Including users in work/employment support programs, regardless of the severity of their symptoms or other underlying difficulties Conducting a job search as soon as the person shows interest in working Integrating vocational support with clinical care in an individualized way Being aware of users' preferences regarding a job/occupation Identifying and addressing cases of discrimination at work/employment and equivalence between different respondents. 13,16 Test-retest reliability tends to decrease when the time interval is prolonged, and for this reason, we conducted a verification within an interval of 10 to 14 days between the test and retest on a sample of at least 15% of the participants or a minimum of 50 respondents. 26,27 For the verification of internal consistency, values above 0.70 for the α-Cronbach are accepted as adequate. As for the Intraclass Correlation Coefficient, values between 0.6 and 0.79 indicate a substantial correlation and values greater than or equal to 0.80 indicate an almost complete correlation. 14,15,26,27 It is noteworthy that the present study followed all the recom- Precisely determining the aspects related to the instrument's validity and reliability measures allows for greater assurance of the quality of results. However, it is worth clarifying that validity and reliability are not fixed measures and may vary according to the population, type of study, and purpose.

CONCLUSION
Through the values obtained in the validation process of The Providers Survey instrument using the Consensus-Based Standards for the Selection of Health Measurement Instruments checklist, we found that the Brazilian version of the instrument had content validity, construct validity, and reliability, which was verified by internal consistency and stability. Therefore, it is an instrument capable of exploring the phenomenon to be studied, and its items reliably and adequately represent the measured construct. All the instrument's parts measure the same characteristic, which guarantees its reliability, and confers homogeneity among the different respondents.
The instrument developed instrument was named the Work Assessment Instrument for Recovery in Mental Health (IATRE-SM).
From its use, it is expected that mental health services in Brazil will be guided by the Recovery concept to prepare users so that they can face society and engage in the recovery process. It is also Encouraging the person to seek employment Identifying and addressing negative internalized views of themselves that make people believe they are incapable of working Involving the family in supporting the user's efforts to seek or maintain employment Engaging peers in supporting the user's efforts to seek or maintain work/job Recognizing the work/job as important in the recovery of the user Recognizing the job as a source of stress that should be avoided Recognizing employment as offering a valuable social role or as an important source of identity reinforcement Recognizing employment as a risk factor for relapse/crisis of users in mental suffering PART 3 Indicate the degree of importance of the following components to promote the restoration/recovery of users in mental suffering. Where: 1-Not important, 2-Somewhat important, 3-Not at all and not very important, 4-Important, 5-Very important, 6-I prefer not to answer. DOMAIN 1 2 3 4 5 6 Being hopeful Having a life project Having stable housing/place Being abstinent from drugs and alcohol Having family support Having friends or people to trust and/or partner or spouse Adhering to prescribed treatments Having a sense of belonging in the community and value their cultural and social identity Taking control of their own life/autonomy Having something that gives meaning to life Having quality medical care and/or multi-professional assistance Believing in oneself as a capable person Participating in recreational/leisure social activities Having a long period of stability (i.e., no crises) Eating healthy and practice physical activity Being valued for their activities in the community Table 4. Continuation expected that users will be perceived as individuals capable of integrating into society and exercising their autonomy, assuming an active role in the community. Based on the results of this study, we encourage research on the perceptions of health professionals regarding the relationship between work and recovery for people with mental disorders.